Method and system for detecting a type of seat occupancy

ABSTRACT

Computer implemented method for detecting a type of seat occupancy, comprising capturing, by means of an imaging device, an image of a seat, the image comprising depth data and intensity data, performing, by means of a processor device, a classifier algorithm on the captured image to determine a level of occupancy, wherein, if the determination indicates that the level of occupancy is above a predetermined threshold, the method comprises processing, by means of the processor device, the depth data with a convolutional neural network, to determine a type of occupation and wherein, if the determination indicates that the level of occupancy is below a predetermined threshold, the method comprises processing, by means of the processor device, the intensity data with a convolutional neural network to determine a type of occupation.

FIELD

The present disclosure relates to methods and systems for detecting atype of seat occupancy, in particular in a vehicle.

BACKGROUND

Digital imaging devices, such as digital cameras, are used in automotiveapplications to detect passenger occupancies in a vehicle.

Thereby, it is important to detect whether a seat is occupied orunoccupied and in particular what type of occupation is present on theseat.

Accordingly, there is a need for improved methods and systems fordetecting a seat occupancy.

SUMMARY

The present disclosure provides a computer implemented method, acomputer system and a non-transitory computer readable medium accordingto the independent claims. Embodiments are given in the subclaims, thedescription and the drawings.

In one aspect, the present disclosure is directed at a computerimplemented method for detecting a type of seat occupancy, comprisingcapturing, by means of an imaging device, an image of a seat, the imagecomprising depth data and intensity data, performing, by means of aprocessor device, a classifier algorithm on the captured image todetermine a level of seat occupancy. If the determination indicates thatthe level of seat occupancy is above a predetermined threshold, themethod comprises processing, by means of the processor device, the depthdata with a convolutional neural network, to determine a type ofoccupation. If the determination indicates that the level of seatoccupancy is below the predetermined threshold, the method comprisesprocessing, by means of the processor device, the intensity data with aconvolutional neural network to determine a type of occupation.

The method is in particular suitable for detecting a type of seatoccupancy in a vehicle. Therein, the vehicle comprises an imagingdevice, that is adapted to capture one or more images, in particular ofthe passenger compartment of the vehicle.

The imaging device may be, for example, located on the inside of theroof of the vehicle, in particular the roof liner, and covering at leasta portion of the passenger compartment comprising at least one seat. Inparticular, the imaging device covers the entire passenger compartmentwith all seats present therein, in a top view or bird's eye view.Particularly there is only a single imaging device provided.

Therein, the imaging device is adapted to capture one or more imagescomprising depth data and intensity data. The imaging device may be, forexample, a time-of-flight camera, a stereo camera or a radar camera. Thedepth data represent, for each pixel, a depth information, whichcorresponds to a real distance in height, and the intensity datarepresent a luminescence for the corresponding pixel.

Then, a classifier algorithm is processed on the captured image todetermine a level of seat occupancy. A level or a degree of seatoccupancy may refer to a value how much, in particular how much of thearea of the seat is occupied. The classifier algorithm may use inparticular only the depth data to determine a level of seat occupancy.This may be done, for example, by comparing a height profile derivedfrom the depth data with a reference height profile to detectdifferences in the captured height profile. The level of seat occupancymay correspond to a degree of deviation of the captured height profilefrom the reference height profile of the seat.

If the determination indicates that the level of seat occupancy is abovea predetermined threshold, the method comprises processing, by means ofthe processor device, the depth data with a convolutional neuralnetwork, to determine a type of occupation. A type of occupation in thiscase means, for example, to determine whether a person or a largeobject, such as a child seat, is present on the seat.

If the determination indicates that the level of seat occupancy is belowthe predetermined threshold, the method comprises processing, by meansof the processor device, the intensity data with a convolutional neuralnetwork to determine a type of occupation. A type of occupation in thiscase means, for example, that the seat is fully empty or only occupiedby a small item, but not a person or a child seat.

The predetermined threshold of seat occupancy may be set at 50% of seatoccupancy. Alternatively, it may be set at 30% of seat occupancy.Further alternatively, it may be set at 10% seat occupancy. Inparticular, the threshold may correspond to a certain object size, forexample an object that is not larger than 5 cm, 10 cm or 15 cm in width,height and/or depth may be considered as being below the threshold and acorrespondingly larger item may be considered as being above thethreshold.

Thereby, the two data channels comprising of depth data and intensitydata are being used according to their potential. In particular, depthdata are particularly suitable to determine an occupancy by a person ora large object while intensity data are particularly suitable todistinguish whether the seat is empty or if a small item is present onthe seat.

Through the method it is possible to ignore unnecessary parts of thedata and only subject the respective parts of the imaging data to therelevant algorithms. Further, an information overhead may disturb theresults of the convolutional neuronal network, which is avoided.Further, computational resource requirements are reduced.

The method may be repeated periodically or in particular upon startingof an engine, shutting down an engine, unlocking a vehicle or locking avehicle.

According to an embodiment, the method further comprises performing, bymeans of the processor device, a resizing algorithm on the depth dataand/or the intensity data.

In particular, the depth data and/or the intensity data may be resizedto be particularly well processable by the convolutional neural network.

According to an embodiment, the classifier algorithm is adapted todetermine an estimated seat distance from the image. Therein, theestimated seat distance is used to resize intensity data.

In particular, by deriving the seat distance, which is particularlypossible based on the depth data, it is particularly easy to resize theimage and in particular to resize the intensity data contained in theimage.

According to an embodiment, the method further comprises performing, bymeans of the processor device, a smoothing algorithm on the depth data.

A smoothing algorithm may, for example, be a median filter. By smoothingthe depth data, unwanted noise, which is usually present in depth datamore than in intensity data, is filtered out, leading to better results.

According to an embodiment, the method further comprises performing, bymeans of the processor device, a crop algorithm on the depth data and/orthe intensity data.

The crop algorithm may in particular be performed before the resizingand or smoothing algorithm. By cropping the respective data to only therelevant seat area, computational resources are preserved.

According to an embodiment, performing a crop algorithm on the depthdata comprises performing a static crop algorithm of the depth data.

The depth data are cropped with a static factor as this channel isparticularly provided to distinguish whether there is a person or achild seat occupying the seat, which has to take into accountsurroundings of the seat, in particular parts of the seat that not onlycomprise the bottom rest area. Thus, the cropping algorithm is veryquick.

According to an embodiment, performing a crop algorithm on the intensitydata comprises performing a dynamic crop algorithm of the intensitydata.

In contrast to the depth data, the intensity data are particularlyprovided to distinguish whether the seat is totally unoccupied or ifthere is a small item present on the seat. Therefore, it is helpful todynamically crop the intensity data image, to be able to identify alsosmall objects.

According to an embodiment, the classifier algorithm is adapted todetermine an estimated seat region from the image. Therein, theestimated seat region is used to dynamically crop the intensity data.

In particular, by deriving the seat region, which is particularlypossible based on the depth data, it is particularly easy to determinethe exact location on which a small object may have been placed on theseat, in particular the bottom rest in the image.

According to an embodiment the convolutional neural network processingthe intensity data is a Siamese convolutional neural network.

Such a Siamese convolutional neural network is particularly suitable todistinguish between an unoccupied seat and a small object being placedon the seat.

According to an embodiment the convolutional neural network processingthe intensity data uses a first reference image of an unoccupied seat todetermine a type of occupation.

The reference image may be a previously captured image of the same seatin an unoccupied status.

The use of a reference image works particularly well with a Siameseconvolutional neural network.

According to an embodiment, the method further comprises defining, bymeans of the processor device, a previously captured image as a secondreference image, if it has been determined for a first predeterminednumber of times that that the level of seat occupation is above thepredetermined threshold, in particular that the seat is occupied.

In particular, if it has been determined that the level of seatoccupation is above the predetermined threshold for, for example 5times, 10 times or 20 times, in particular in a row, without determiningthat the level of seat occupation is below the predetermined threshold,this might be an indicator that the seat color has changed, for exampledue to discoloring subject to sunlight or because of a stain, such thatthe first reference image is not similar enough to the captured imagefor the convolutional neural network.

In this case, adding a second reference image to the first referenceimage, with or without replacing the first reference image, improves thedetection probability.

The second reference image may in particular be the last capturedreference image for which is has been determined that the level of seatoccupation is below the predetermined threshold. Therefore, capturedimages are regularly stored in a memory device. Thus, the secondreference image is particularly recent.

According to an embodiment, defining a previously captured image as asecond reference image comprises processing, by means of the processordevice, a plurality of previously captured images with a convolutionalneural network to determine a most relevant previously captured image tobe defined as a second reference image.

In this particular embodiment, multiple images from both, adetermination that the level of seat occupation is below thepredetermined threshold and a determination that the of seat occupationis above the predetermined threshold are used together and compared todetermine one most relevance captured image.

According to an embodiment, the method further comprises requesting, bymeans of the processor device, a user to unoccupy the seat, capturing,by means of the imaging device, an image of the unoccupied seat anddefining, by means of the processor device, the captured image of theunoccupied seat as a third reference image.

The user is requested to unoccupy the seat, i.e. to remove items fromthe seat and clear it. The request may for example be output on aninfotainment device of the vehicle. Thereby, it is possible to have areference image that is most up to date. The third reference image maybe used with or without replacing previous reference images.

In another aspect, the present disclosure is directed at a computersystem, said computer system being configured to carry out several orall steps of the computer implemented method described herein.

The computer system may comprise a processor device, at least one memorydevice and at least one non-transitory data storage device. Thenon-transitory data storage device and/or the memory device may comprisea computer program for instructing the computer to perform several orall steps or aspects of the computer implemented method describedherein.

In another aspect, the present disclosure is directed at anon-transitory computer readable medium comprising instructions forcarrying out several or all steps or aspects of the computer implementedmethod described herein. The computer readable medium may be configuredas: an optical medium, such as a compact disc (CD) or a digitalversatile disk (DVD); a magnetic medium, such as a hard disk drive(HDD); a solid state drive (SSD); a read only memory (ROM), such as aflash memory; or the like. Furthermore, the computer readable medium maybe configured as a data storage that is accessible via a dataconnection, such as an internet connection. The computer readable mediummay, for example, be an online data repository or a cloud storage.

The present disclosure is also directed at a computer program forinstructing a computer to perform several or all steps or aspects of thecomputer implemented method described herein.

DRAWINGS

Exemplary embodiments and functions of the present disclosure aredescribed herein in conjunction with the following drawings, showingschematically:

FIG. 1 an embodiment of a computer system for detecting a type of seatoccupancy;

FIG. 2 a flow chart of an embodiment of a method for detecting a type ofseat occupancy; and

FIG. 3 a flow chart of another embodiment of a method for detecting atype of seat occupancy.

DETAILED DESCRIPTION

FIG. 1 depicts an embodiment of a computer system 10 for detecting atype of seat occupancy.

The computer system 10 comprises a processor device 11, an imagingdevice 12 and a memory device 13.

Therein, the computer system 10 is configured to capture, by means ofthe imaging device 12, an image of a seat, the image comprising depthdata and intensity data, and to perform, by means of the processordevice 11, a classifier algorithm on the captured image to determine alevel of seat occupancy.

Therein, if the determination indicates that the level of seatoccupation is above the predetermined threshold, the computer system 10is configured to process, by means of the processor device 11, the depthdata with a convolutional neural network, to determine a type ofoccupation.

If the determination indicates that the level of seat occupation isbelow the predetermined threshold, the computer system 10 is configuredto process, by means of the processor device 11, the intensity data witha convolutional neural network to determine a type of occupation.

The computer system 10 is further configured to perform, by means of theprocessor device 11, a resizing algorithm on the depth data and/or theintensity data.

Therein, the classifier algorithm is adapted to determine an estimatedseat distance from the image and the estimated seat distance is used toresize intensity data.

The computer system 10 is further configured to perform, by means of theprocessor device 11, a smoothing algorithm on the depth data.

The computer system 10 is further configured to perform, by means of theprocessor device 11, a crop algorithm on the depth data and/or theintensity data.

Therein, performing a crop algorithm on the depth data comprisesperforming a static crop algorithm of the depth data and performing acrop algorithm on the intensity data comprises performing a dynamic cropalgorithm of the intensity data.

The computer system 10 is further configured in that the classifieralgorithm is adapted to determine an estimated seat region from theimage and the estimated seat region is used to dynamically crop theintensity data.

The computer system 10 is further configured in that the convolutionalneural network processing the intensity data is a Siamese convolutionalneural network.

The computer system 10 is further configured in that the convolutionalneural network processing the intensity data uses a first referenceimage of an unoccupied seat to determine a type of occupation.

The computer system 10 is further configured to define, by means of theprocessor device 11, a previously captured image as a second referenceimage, if it has been determined for a first predetermined number oftimes that that the level of seat occupation is below the predeterminedthreshold.

The computer system 10 is further configured in that defining apreviously captured image as a second reference image comprisesprocessing, by means of the processor device 11, a plurality ofpreviously captured images with a convolutional neural network todetermine a most relevant previously captured image to be defined as asecond reference image.

The computer system 10 is further configured to request, by means of theprocessor device 11, a user to unoccupy the seat, to capture, by meansof the imaging device 12, an image of the unoccupied seat and to define,by means of the processor device 11, the captured image of theunoccupied seat as a third reference image.

FIG. 2 depicts a flow chart of an embodiment of a method 100 fordetecting a type of seat occupancy.

The method 100 starts at step 101 where an image of a seat is captured,wherein the image comprises depth data and intensity data.

In a next step 102, a classifier algorithm is performed on the capturedimage to determine a level of seat occupancy.

If the determination in 102 indicates that the level of seat occupancyis below the predetermined threshold, the method continues with step 103to proceed along the upper path, based on the depth data.

If the determination in 102 indicates that the level of seat occupationis below the predetermined threshold, the method 100 continues with step104 to proceed along the lower path, based on the intensity data.

Following the upper path in FIG. 1 , in a next step 103, a static cropalgorithm of the depth data is performed.

Then, in a next step 105, a smoothing and resizing algorithm on thedepth data is performed.

In a further step 107, the depth data are processed with a convolutionalneural network to determine a type of occupation. This step 107 may leadto the determination 110 that a person is present on the seat.

Alternatively, the step 107 may lead to the determination 111 that achild seat is present on the seat.

Further alternatively, the step 107 may lead to the determination 112,that an object is present on the seat, in particular a large object.

Following the lower path in FIG. 1 , in a next step 104 following step102, a dynamic crop algorithm of the intensity data is performed byusing an estimated seat region 102 a from the classifier in step 102.

Then, in a next step 106, a resizing algorithm on the intensity data isperformed using an estimated seat distance 102 b from the classifier instep 102.

In a further step 108, the intensity data are processed with a Siameseconvolutional neural network to determine a type of occupation by usinga reference intensity image 109 of an empty seat.

This step 108 may lead to the determination 112 that, despite the levelof seat occupancy being below the predetermined threshold, an object isactually present on the seat, in particular a small object.

Alternatively, the step 108 may lead to the determination 113 thatnothing is present on the seat, i.e. the seat is in fact unoccupied.

FIG. 3 depicts a flow chart of another embodiment of a method 200 fordetecting a type of seat occupancy.

In particular, the method 200 as shown in FIG. 3 depicts the selectionof a reference image as used in step 109 in FIG. 2 .

In particular, the first reference image as used in step 109 may beprerecorded and preprocessed, for example, at the end of production ofthe vehicle. However, during the life cycle of the car, an appearance ofthe empty seat may change, either continuously, by attenuation of color,or abruptly, by stains or an applied sating mat.

Therefore, the comparison of step 109 is extended by the method as shownin FIG. 3 and as described in the following.

The standard mode, or default mode, is in step 201, in which one or morereference images are used and the performance is normal. If, however, atsome point, no unoccupied seat has been determined for a firstpredetermined period of time or a first predetermined number of times,it is assumed in path 201 a, that the appearance of the unoccupied seathas changed and the method 200 transitions to step 202, which is calledredefine mode.

In this redefine mode 202, the system attempts to find a new referenceimage in recent images. Therein, in order to timely react to smallappearance changes, a threshold is defined which defines the maximumdistance for empty seats.

Thereby, recent images are compared in order to find similar images atwidely disparate points in time as a clear indication for an unoccupiedseat, as even the same object may not be positioned in the exact samelocation.

For this purpose, captured images are regularly stored during thedefault mode, i.e. in normal operation. To save resources, only suchcaptured images may be stored in which large variations of distanceestimations with respect to the reference image occur or after a changein detection type.

By using the Siamese convolutional neuronal network, the stored imagesare then compared with each other and images with distances below theabove-mentioned threshold are clustered together. These cluster may thenbe analyzed by the number of images and the interval of first and lastoccurrence.

If for both criteria predetermined requirements are fulfilled, forexample at least 3 images and an interval of at least 10 differentdeterminations, the image with the smallest mean distance to the otherimages of the cluster is chosen as the most relevant one and used as areference image.

If no cluster fulfills the requirements, the system remains in redefinemode 202 until such a cluster is found or alternatively an unoccupiedseat is detected and then follows path 202 a back to default mode 201.

If this does not happen for a second predetermined period of time or asecond predetermined number of times, the system changes from redefinemode 202 along path 202 b to unknown or HMI mode 203.

In this unknown mode 203, the user is requested to clear the seat,whereupon an image is captured, which is then taken as a referenceimage, transitioning along path 203 a back to default mode 201.

REFERENCE NUMERAL LIST

-   10 computer system-   11 processor-   12 imaging device-   13 memory device-   100 method-   101 method step-   102 method step-   102 a method step-   102 b method step-   103 method step-   104 method step-   105 method step-   106 method step-   107 method step-   108 method step-   109 method step-   110 method step-   111 method step-   112 method step-   113 method step-   200 method-   201 method step-   202 method step-   203 method step

1. Computer implemented method for detecting a type of seat occupancy,the method comprising: capturing, by means of an imaging device, animage of a seat, the image comprising depth data and intensity data;performing, by means of a processor device, a classifier algorithm onthe captured image to determine a level of seat occupancy; if thedetermination indicates that the level of seat occupancy is above apredetermined threshold: processing, by means of the processor device,the depth data with a convolutional neural network, to determine a typeof occupation; and; if the determination indicates that the level ofseat occupancy is below the predetermined threshold: processing, bymeans of the processor device, the intensity data with a convolutionalneural network to determine a type of occupation.
 2. Computerimplemented method according to the previous claim 1, further comprisingperforming, by means of the processor device, a resizing algorithm onthe depth data and/or the intensity data.
 3. Computer implemented methodaccording to the previous claim 2, wherein the classifier algorithm isadapted to determine an estimated seat distance from the image; andwherein the estimated seat distance is used to resize intensity data 4.Computer implemented method according to claim 1, further comprisingperforming, by means of the processor device, a smoothing algorithm onthe depth data.
 5. Computer implemented method according to claim 1,further comprising performing, by means of the processor device, a cropalgorithm on the depth data and/or the intensity data.
 6. Computerimplemented method according to the previous claim 5, wherein performinga crop algorithm on the depth data comprises performing a static cropalgorithm of the depth data. (Currently Amended) Computer implementedmethod according to claim 5, wherein performing a crop algorithm on theintensity data comprises performing a dynamic crop algorithm of theintensity data.
 8. Computer implemented method according to the previousclaim 7, wherein the classifier algorithm is adapted to determine anestimated seat region from the image; and wherein the estimated seatregion is used to dynamically crop the intensity data.
 9. Computerimplemented method according to claim 1, wherein the convolutionalneural network processing the intensity data is a Siamese convolutionalneural network.
 10. Computer implemented method according to claim 1,wherein the convolutional neural network processing the intensity datauses a first reference image of an unoccupied seat to determine a typeof occupation.
 11. Computer implemented method according to the previousclaim 10, further comprising: if it has been determined for a firstpredetermined number of times that that the level of seat occupancy isbelow the predetermined threshold: defining, by means of the processordevice, a previously captured image as a second reference image. 12.Computer implemented method according to the previous claim 11, whereindefining a previously captured image as a second reference imagecomprises: processing, by means of the processor device, a plurality ofpreviously captured images with a convolutional neural network todetermine a most relevant previously captured image to be defined as asecond reference image.
 13. Computer implemented method according toclaim 10, further comprising: requesting, by means of the processordevice, a user to unoccupy the seat; capturing, by means of the imagingdevice, an image of the unoccupied seat; and defining, by means of theprocessor device, the captured image of the unoccupied seat as a thirdreference image.
 14. Computer system, the computer system beingconfigured to carry out the computer implemented method of claim
 1. 15.Non-transitory computer readable medium comprising instructions forcarrying out the computer implemented method of claim 1.